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Acquisition  Problems  Supporting  Data 


♦  Current  Acquisition  Strategy  Reported  Results: 

-  YR2000:  84%  of  programs  are  late  and  over  budget,  and 
deliveries  include  only  61%  of  planned  capabilities* 


-  YR2004:  40%  ($8  Billion)  of  DoD  RDT&E  Budget  was  spent  on 
reworking  software  due  to  quality  issues** 


-  YR2009:  DOB’S  95  major  defense  acquisition  programs  have 
seen  their  costs  grow  by  an  average  of  26%  and  experienced  an 
average  schedule  delay  of  almost  2  years*** 


Program  Offices  are  Failing  to  Successfully  Scope  and  Manage 

SW Intensive  Procyams 


*  2000  Defense  Science  Board  (DSB)  Task  Force  on  Defense  Software  Report 
**  2004  General  Accountability  Office  Report 

***  2009  Opening  Statement  of  Senator  Carl  Levin  at  Senate  Armed  Services  Committee  Hearing,  March  3,  2009 
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Problem 


♦  Software  size,  complexity,  and  reliance  is  continuing  to 
significantly  expand  within  DoD/Navy  critical  systems 

♦  DoD/Navy  is  Failing  to  Consistently  Successfully  Acquire 
Software  Intensive  Systems 

-  7  Key  Acquisition  Management  Problems  Exist* 

1 .  Lack  of  Effective  Acquisition  Management 

2.  Immature  Acquirer  (Program  Offices) 

3.  Ineffective  Requirements  Management 

4.  High  Personnel  Turnover  in  the  Acquiring  Organizations 

5.  Cost  and  Schedule  Estimation  Accuracy 

6.  Ineffective  Utilization  of  EVMS  for  SW Systems 

7.  Failure  to  Take  Advantage  of  Lessons  Learned 

*  2007  ASN  /  RDA  Software  Process  Improvement  Initiative  (SRI)  As-ls  Report  for  SW  Acquisition  Management 


Loss  of  Government  In-house  Applied  SW  Expertise 


Current  State 

Software  Acquisition  Strategy 
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DoD/ASN/RDA  Policies  Call  for  Gov’t  SMEs  to  Define  System  Req’s,  Support  Milestone  Reviews,  and  Validate  the  SW  Artifacts  Developed  by  Industry 

Software  Development  Activities  Conducted  Primarily 
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Recent  Findings  &  Recommendations 


♦  2008  GAO  Report 

-  Increased  and  Improved  Government  Oversight  is  Required 

♦  2008  DSB  DTE  Report 

-  Key  Factor  is  High  %  of  Programs  Failing  IOTE  is  Loss  of 
Experienced  Management  and  Technical  Personnel 

♦  2008  SECNAV  Memo 

-  DoD  Must  Maintain  Technical  Expertise  at  all  Levels 

♦  Informal  on  site  visits  and  discussions  with  several 
VNferfare  Center  Software  Leads  indicates  that  the  majority 
of  the  critical  software  development  is  being  contracted 
out  to  private  Industry 


Government  Needs  to  Reconstitute  In-House  Technical  Expertise 
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DOD/Navy  Software  Acquisition 
Current  State  Summary 


Gov’t 

Expertise 


Size  & 
Complexity 


Acquisition 

Failures 


Time:  Past  20  years  of  DOD/Navy  SW  Acquisition 


“The  combination  of  personnel  reductions  and  reduced  RDT&E  has  seriously  eroded,  the  Departments  domain  knowledge 
anc[  produced  an  over  reliance  on  contractors  to  perform  core  in-house  technical  functions.” 

“In  order  to  acquire  the  DON  platforms  and  weapons  systems  in  a  responsible  manner,  it  is  imperative  the  DON 
maintain  technical  domain  expertise  at  al[  [ evels  of  the  acquisition  infrastructure 
2008,  Oct  10,  SECNAV  MEMO:  Department  of  the  Navy  Acquisition,  D.  Winter 
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Conclusions  /  Imperatives 


♦  The  Government  Must  Maintain  Applied  Software  Development 
Expertise  in  order  to  be  a  Smart  Buyer  of  SW  intensive  Systems 

♦  There  Must  be  a  Fair  Balance  Struck  Between  all  of  the  Following: 

-  Industry’s  Right  and  Requirement  to  Make  a  Fair  and  Deserved  Profit 

-  Government’s  Responsibility  to  Provide  the  Whr  Fighter  VMth  the  Highly 
Reliable,  Safe,  and  Adaptive  Systems 

-  Government’s  Responsibility  to  Spend  the  Tax  Payers  Dollars  Effectively  and 
Efficiently 


It  is  Imperative  that  the  Gov’t  Maintain  the  In-house  Applied  SW 
Technical  Expertise  Required  to  Successfully  Acquire  SW  Intensive 

Systems 
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Proposed  Software  Acquisition  Strategy 
Future  State 


Gov’t 

Expertise 


Size  & 
Complexity 


Acquisition 

Failures 


Reconrrnendations 

1.  Reconstitute  the  Navy’s  in-house  applied  sw  development  expertise;  establish  sw  pipe-line 

2.  Utilize  government  and  industry  software  development  Integrated  Product  Teams 
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Future  State:  Strategy  Recommendation 
Reconstitute  Navy  In-House  Software  Expertise  Pipe-Line 


H 

I 

G 

H 


Line  and  Program  Management  Path 


Technical  Path 


Challenge  2 

Maintaining  Navy  in-house  SW expertise  requires  that 
an  appropriate  subset  of  critical  SW  be  developed  in- 
house.  There  is  no  defined  criteria  or  process  for 
assigning  sw  development  to  in-house  engineers. 


Challenge  1 

Non  SW  Background 


Management:  System(s)  Level 
Division  (100  to  250)  Head 
Wbrfare  Center  Program  Manager 


Domain  and  System  Level  Leadership 

Program  &  Line  Management: 
Department  Head  (500+) 

Program  Managers  for  PEOs 


Management:  Component  Level 
Branch  (25  to  40)  Head 


Management:  CSCI(s)  Level 
SW Group  (4  to  10)  Leadership 


Technical  Leadership  and  Oversight 
Systems  and  Domain  Level 
-  AoA  Leadership  and  Execution 
-Cost  and  Schedule  Assessment 
-Tech  Approach  Leadership  &  Approval 


‘ TECHNICAL  ASSIGNMENT  LOOP-BACK’ 


Segment  and  Component  Level  Development 
Lead  Architecture  Design  and  Implementation 

-  Cross  Organization/Function  IPT  Leadership  and  Participation 

-  Lead  Technology  insertion 


CSC  Level 
Small  Changes 


Computer  SW  Configuration  Item  (CSCI) 
Lead  CSCI  Architecture  Design  and  Code 

-  Cross  Discipline  IPT  participation 

-  Complex  Tech  Problem  Resolution  ■ 


Senior  Level  SW  Experts 


1  to  2  years 


2  to  8  years 


8  years 


Time 


“in  order  to  acquire  the  DON  platforms  and  weapons  systems  in  a  responsible  manner,  it  is  imperative  the  DoN 
maintain  technical  domain  expertise  at  all  levels  of  the  acquisition  infrastructure”. 

-Department  of  the  Navy  Acquisition,  D.  Writer:  SECNAVMemo  Dated  10  Oct  08 
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Future  State:  Strategy  Recommendation 

Gov’t  In-House  Software  Expert  Responsibilities 


♦  Ownership  of  the  Objective  Architecture 

-  Determine  which  SW Components  Should  be  Reused,  Modified,  or  Developed 

♦  Developing  a  Subset  of  the  Critical  and  Complex  SW  Components 

-  Maintain  Expertise  with  Complex  System  Functionality, 

-  Maintain  Expertise  with  Latest  Software  Development  Technologies  and 
Methodologies 


♦  Leading  Integrated  Gov’t  and  Industry  SW  Development  Teams 

-  In-House  SW  Experts  have  SW  Design  Technical  Approval  Authority 

-  Ensure  SW  meets  Open  Architecture  objectives 

-  Ensuring  Industry  Adheres  to  Best  SW  Development  Practices 


The  Proposed  Government  And  Industry  Software  Teaming  Strategy  is  already  being 
Successfully  Utilized  for  a  Some  Critical  Fire  Control  Systems 
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Work  Share 


Future  State 

SW  Development  Responsibility  Allocation 


♦  Industry  will  still  develop  the  majority  of  the  software 

♦  Government  in-house  SW  Experts  will  provide  more  SW  Leadership  and  Authority 


Industry  will  still  develop  a  majority  of 
the  SW;  but  with  Gov’t  Software  SME 
oversight  and  insight 


_ I _ A _ 
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Future  State  Goals 

Software  Evolution  to  Open  Architecture  (OA) 


CURRENT :  Stove  Pipes 


Non-QA 

SW 


Platform  “1”  Unique  Development’  SW  growth 


Limited  SW  Re-use 
between  systems/platforms 


Non-QA 

SW 

Non-QA 

SW 

^  _ . 

^  1 

Platform  “N”  Unique  Development’  SW  growth 


Few  Open  Arch  based  designs  and  software 


FUTURE:  OA  Based  Multi -Ratform  Capable 


Open  Arch 
Reusable  SW 
Components 


■ 


New  Capability 
NewQASW  [ 
Components 


Modified  SW 


Components 


Open  Arch 


Reusable  SW 
Components 


New  Capability 

NewQASW 

Components 


MocEfiedSW 
■  Components 

:  ;|  Open  Arch 
_  Reusable  SW 
Components 


L 

n 


SW  stored  in  Gov’t  Owned  Common  SW  Repository 


Establish  truly  QA  based  reusable,  scalable,  modular, 
and  maintainable  components 
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Future  State 

Challenge:  Open  Architecture  Software 


Composability 

The  System  Provides  Recombinant 
Components  that  can  be  Selected 
and  Assembled  in  Various  Combinations 
to  Satisfy  Specific  Requirements 


\ 


Reusability 


iC 


Maintai  liability 

The  Ease  With  Which  Maintenance  of 
a  Functional  Unit  can  be  Performed  in 
Accordance  With  Prescribed  Requirements 


Interoperability 

Ability  of  Two  or  More  Subsystem 
to  Exchange  Information  and  Utilize 
that  Information 


Ability  for  an  Artifact  to  Provide 
the  Same  Capability  in 
Multiple  Contexts 


Extensibility 

Ability  to  add  new  Capabilities  to  System 
Components,  or  to  add  Components 
and  Subsystems  to  a  System 


These  OA  “IUTIES”  Cannot  be  Easily  \ferified  by  System  Testing. .  Government  In-House  SW 
Expertise  Insight  into  i Design  and  Code  is  Required  to  Ensure  Reusable  Software 

Designing  and  Coding  for  These  “IUTIES”  is  the  Key  to  Saving  Significant  $$$$$$$$ 


*  Reference:  OA  Architectural  Principles  and  Guidelines  v  1.5.6, 2008,  IBM,  Eric  M  Nelson,  Acquisition  Community  Website  (ACC)  DAU  Navy  OAVNfebsite 
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Current  State  Challenge: 

Levels  of  SW Complexity/ Devil  is  in  the  Details 


SYSTEM 

Functional  Domain 

SW  System  Components 


Gov’t  technical  insight 
only  at  the  Func,  Comp, 
Segment  levels  is  not 
sufficient  to  ensure  & 
meet  OA  goals 
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Gov’t  SWSIVEs  must  ensure  QA  req’s  are  met  at 
the  detailed  SW design  level  for: 

-  Open  Standards  -  Maintainability 

-  Reuse  -  Interoperability 

-  Modularity  -  Composability 

-  Extensibility 

Gov’t  SWSMEs  must  understand  the  technical 
design  for: 

-  Data/ File  Management 

-  Threading /Tasking  Hierarchy 

-  Initialization /Termination 

-  Time  Critical  &  Deterministic 

-  Intra&  Inter  Process  Comrrfs 

-  Fault  Processing 

-  Process  PrioritizationX 
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Common  Hardware  and  Operating  Systems 


III  I 

Asingle  erroneous  SLOC  can  crash  the  entire  system 
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Millions 


Open  Architecture:  Example 

Reusable,  Maintainable,  Scalable  Softv vane  Design 


♦  VNfeapon  System  X 
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Future  State:  Benefits 


♦  By  establishing  the  government  sw expertise  pipe-line;  the  government  will  have 
the  sw  expertise  required  to  address  the  current  state  Acquisition  Challenges  and 

-  Improve  software  technical  approach/maturity  identification  and  assessment 


-  Improve  software  requirements  change  management  and  assessment  of  the  associated 
impacts  to  cost,  schedule,  technical  performance  and  risk 


-  Maintain  system  and  software  architecture  corporate  knowledge  as  program  office 
leadership  turns  over,  and  as  system  development  responsibility  transitions  to  different 
private  industry  contractors 


-  Improve  software  cost  estimation  and  tracking  (EVMS) 

★  Lead  process  improvement  efforts  based  on  applied  experience  and  historical  data 


-  Ensure  QA  based  reliable,  maintainable,  reusable,  scalable,  &  modular  software 
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Summary 


♦  DoD  /  Navy  Systems’  Software  Size  and  Complexity  is  Significantly  Increasing 

♦  Navy  Applied  In-House  SW  Expertise  is  Decreasing 

-  Program  Offices  do  Not  Have  the  Applied  SW  Expertise  Required  to  Consistently 
Successfully  Acquire  SW  Intensive  Systems 

♦  Must  Reconstitute  and  Maintain  the  Navy’s  In-House  SW  Development  Expertise 

-  Must  Maintain  Applied  SW  Expertise  and  Experience  Developing  Complex  SW  With 
Real-Time,  Safety  Critical,  Multi -ThreadecVProcess,  Complex  Interfaces  and  Algorithms 

-  Requires  that  a  Subset  of  the  Complex  and  Critical  Software  be  Developed  In-House 


Developing  Government  SWSNEs  will  Enable  Navy’s  Goal  of  Open  Architected  Systems  and 
Improve  the  Ability  to  Consistently  and  Successfully  Deliver  Systems  That  Meet  Cost, 

Schedule,  and  Technical  Performance  Goals 
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Back-Up 


♦  Current  Acquisition  State  Summary 

♦  References 
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53%  have  cost  growth  over  89% 


The  DOD/Navy  is  not  consistently  successfully  acquiring  software  intensive  systems. 
The  DOD/Navy  needs  to  reconstitute  its  in-house  applied  sw  expertise 
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Report  of  the  Defense  Science  Board  Task  Force  on  Defense  Software,  November  2000  Office  of  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology 

*  84%  of  program  do  not  complete  on  budget  nor  schedule;  31%  are  canceled;  remaining  53%  have  cost 
growth  exceeding  89%;  final  product  only  includes  61%  of  planned  features 

2004  General  Accountability  Office  released  a  report  describing  the  results  of  a  study  to  identify  the  practices 
used  by  leading  companies  to  acquire  software  and  to  analyze  the  causes  of  poor  outcomes  of 
selected  DOD  programs.  GAO  reported  : 

“In  recent  years,  DOD  has  attributed  significant  cost  and  schedule  overruns  of  software-intensive  systems  to 
difficulties  in  developing  and  delivering  software.  DOD  estimates  that  it  spends  about  40  percent  of  its  Research, 
Development,  Test,  and  Evaluation  budget  on  software — $21  billion  for  fiscal  year  2003.  Furthermore,  DOD  and 
industry  experience  indicates  that  about  $8  billion  (40  percent)  of  that  amount  may  be  spent  on  reworking 
software  because  of  quality-related  issues.”  (GAO.  Stronger  Management  Practices  are  Needed  to  Improve  DOD  s 
Software-Intensive  Weapon  Acquisitions .  GAO-04-393.  March  2004) 

2007  SPII  Software  Acquisition  Management  (SAM)  Team  “As-Is  State”  Report 
seven  consistent  primary  problems: 

1 .  Lack  of  effective  acquisition  management 

2.  Immature  acquirer  is  challenged  to  assess  developer  performance 

3.  Ineffective  requirements  management 

4.  High  personnel  turnover  in  the  acquiring  organization 

5.  Cost  and  schedule  estimation  accuracy 

6.  Ineffective  utilization  of  EVMS  for  software  intensive  acquisition  programs. 

7. Failure  to  take  advantage  of  Best  Practices  and  Lessons  Learned. 


Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


21 


♦  Report  of  the  DSB  Task  Force  on  Development  Test  and  Evaluation,  May 
2008 


-  “ loss  of  a  large  number  of  the  most  experienced  management  and  technical  personnel 

..without  an  adequate  replacement  pipeline ”  is  one  of  the  key_  contributors  to  the  trend 
of  a  high  percentage  of  DOD  system  operationally  effective  and  suitability  failures 

-  “over  time,  in-house  DOD  offices  of  subject  matter  experts  were  drastically  reduced, 
and  in  some  cases,  disestablished 


♦  2009  Opening  Statement  of  Senator  Carl  Levin  at  Senate  Armed  Services 
Committee  Hearing  on  DOD  Acquisition  of  Major  weapon  Systems,  March 
3,  2009 

-  DOD’s  95  major  defense  acquisition  programs  have  seen  their  costs  grow  by  an 
average  of  26%  and  experienced  an  average  schedule  delay  of  almost  2  years 


♦  OA  Architectural  Principles  and  Guidelines  v  1.5.6,  2008,  IBM,  Eric  M. 
Nelson,  Acquisition  Community  Website  (ACC)  DAU  Navy  OA  Website 
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Acronyms 


♦  ASN/RDA:  Assistant  Secretary  of  the  Navy  Research  Development  and  Acquisition 

♦  ASR:  Alternative  System  Review 

♦  CDR:  Critical  Design  Review 

♦  CSC:  Computer  Software  Component 

♦  CSCI:  Computer  Software  Configuration  Item 

♦  CMMI:  Capability  Maturity  Model  Integration 

♦  DoD:  Department  of  Defense 

♦  DSB:  Defense  Science  Board 

♦  EVMS:  Earned  Value  Management  System 

♦  FQT:  Formal  Qualification  Test 

♦  GAO:  Government  Accounting  Office 

♦  GOVT:  Government 

♦  IEEE:  Institute  of  Electrical  and  Electronics  Engineers 

♦  ITR:  Initial  Technical  Review 

♦  OA:  Open  Architecture 

♦  OTRR:  Operational  Test  Readiness  Review 

♦  PCR:  Physical  Configuration  Review 

♦  PDR:  Preliminary  Design  Review 

♦  PRR:  Production  Readiness  Review 

♦  RDT&E:  Research  Development  Test  and  Engineering 

♦  SFR:  System  Functional  Review 

♦  SLOC:  Source  Lines  of  Code 

♦  SME:  Subject  Matter  Experts 

♦  SPII:  Software  Process  Improvement  Initiative 

♦  SRR:  System  Requirements  Review 

♦  SVR:  System  Verification  Review 

♦  SW:  Software 

♦  TRR:  Test  Readiness  Review 

♦  WCS:  Weapon  Control  System 
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